1. Onboard Camera
A vehicle is increasingly generally equipped with a camera eliminating blind spots from the driver's position and improving visibility. It is said that the number of shipped cameras for visual recognition exceeds the two-million mark in 2006. The domestic sales number of new cars including trucks is a little less than six millions a year, and hence it follows that every third vehicle is mounted with a camera for visual recognition.
Cameras for visual recognition include (1) a rear camera displaying an image at the rear of the vehicle when the vehicle is reversed, (2) a front-side camera displaying an image of the front wheel portion opposite to the driver, (3) a front camera for visually recognizing a blind spot immediately in front of the vehicle, and (4) a side-blind camera displaying an image of a side blind spot formed when the vehicle is started from a narrow alley or the like. At present, the rear camera occupies 77% of the cameras for visual recognition and forms the mainstream.
In an onboard camera for visual recognition, a wide-angle optical system is generally employed in order to minimize the range of blind spots. In the rear camera, the angle of a horizontal plane exceeds 100 degrees. Large “barrel” distortion aberration takes place in an image picked up by the onboard camera including such an optical system. Due to this distortion, it is hard for the driver to intuitively recognize the position of an obstacle appearing in the image and a feeling for the distance thereto.
2. Distortion Correction
2.1 Technique of Lens Distortion Correction by Optical Processing
In general, there are two methods for correcting barrel distortion. The first one is a method of suppressing distortion by devising the structure of the optical system, and the second one is a method of digitally correcting an image after picking up the image.
Optical distortion correction is implemented by devising the structure of the optical system with an aspherical lens or the like, as described in Non-Patent Documents 1 and 2.
Non-Patent Document 1: Free Encyclopedia “Wikipedia”, Section of Photographic Lenses
Non-Patent Document 2: Advanced Technique of Digital Cameras, Chapter 4 Lens•Optical System Design, Technical Information Institute, Co., Ltd.
However, distortion aberration increases in proportion to about the cube of the angle of view, and it is difficult in a wide-angle optical system employed for the onboard camera to completely correct the distortion by merely devising the optical system. Further, the degree of correction is decided in the design stage of the optical system, and it is difficult to adjust the degree of distortion on the part of the user. In addition, the aspherical lens is produced with a mold, and hence the initial investment disadvantageously increases.
2.2 Technique of Lens Distortion Correction by Image Processing
As a technique of correcting lens distortion of a camera by image processing, the technique of Tsai (refer to Non-Patent Documents 3 and 4) and the algorithm proposed by Zhengyou Zhang (refer to Non-Patent Documents 5 and 6) are commonly employed. This technique is proposed also as a library packaged as a computer program (refer to Non-Patent Document 7), and can be utilized also for distortion correction in the present invention.
Non-Patent Document 3: R. Y. Tsai: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV camera and lenses; IEEE Journal of Robotics and Automation, RA-3, 4, pp. 323-344 (1987).
Non-Patent Document 4: Computer Vision: Technical Review and Future Prospects, June 1998, Naoki Asada
Non-Patent Document 5: Z. Zhang. “Flexible Camera Calibration By Viewing a Plane from Unknown Orientations”, Proc. Of ICCV '99, pp 666-673, 1999.
Non-Patent Document 6: Z. Zhang. “A Flexible New Technique for Camera Calibration”, IEEE, PAMI 22(11), pp 1130-1134, 2000.
Non-Patent Document 7: OpenCV Reference Manual, 6 3D Reconstruction [Camera Calibration], pp. 6_1-6_3.
According to the provided library, a plurality of images obtained by shooting a regular known pattern present on a plane as shown in FIG. 1 while changing the relative relation between the camera and the plane are employed for calculating four parameters (k1, k2, p1 and p2) for performing distortion correction of the camera. These are parameters for controlling distortion correction level. The distortion correction level increases as the values of these parameters increase.
With these parameters, the association between the coordinate positions (x′, y′) of an acquired image and the coordinate values (x, y) after distortion correction can be obtained through the following expressions (1). In this case, the coordinate positions are expressed in a coordinate system taking the x-coordinates in the horizontal direction and taking the y-coordinates in the vertical direction with the origin set at the center of the image. In the expressions (1), r2=x2+y2.x′=x+x(k1r2+k2r4)+2p1xy+p2(r2+2x2)y′=y(k1r2+k2r4)+2p1xy+p2(r2+2y2)  (1)
FIGS. 2A 2B and 2C are schematic diagrams showing examples of images obtained by performing distortion correction with the aforementioned technique. FIG. 2A shows an original image obtained by shooting the known pattern of FIG. 1, FIG. 2B shows a distortion-corrected total image, and FIG. 2C shows a distortion-corrected cutout image. When a wide-angle camera is employed, barrel distortion generally takes place as in the original image, and correction of the distortion results in an image such as the “distortion-corrected total image” having four stretching portions at the corners of the rectangle. Further, conversion for expanding distorted portions is basically performed, and hence the size of the image is larger than that of the original image.
In general, the “distortion-corrected cutout image” obtained by cutting out a rectangular region (region surrounded by a rectangular frame of broken lines in FIG. 2B) from the “distortion-corrected total image” is frequently employed for display, due to such a determination that these images are reduced in visibility. When the internal rectangular region is cut out from the “distortion-corrected total image”, the four portions at the corners of the image stretching in the distortion correction are discarded, and hence it follows that part of information having been acquired in the original image is not displayed. Therefore, the information content of the whole image is reduced, although the visibility of the observer may be improved. In the distortion correction and the cutout, processing such as enlargement or reduction may accompany in response to the image resolution of the display system.
A method of preparing a correction table (look-up table) previously associating addresses before and after correction and converting the addresses of pixels according to this look-up table is conceivable as means for performing distortion correction with hardware. Distortion has a two-dimensional shape and corrected pixel addresses do not necessarily correspond to uncorrected integral addresses, and hence a frame memory storing input images for one screen and an interpolation arithmetic circuit interpolating the pixels are required, to disadvantageously increase the circuit scale.
Patent Document 1 proposes a technique of performing distortion correction with a line memory and an interpolation circuit for two pixels by performing the distortion correction with respect to the horizontal direction. Patent Document 2 proposes a technique of successively processing the X-axis and the Y-axis of an image in two stages, in order to simplify the structure of an interpolation circuit and avoid lacks on the four corners of the image resulting from distortion correction. In each of these Patent Documents 1 and 2, the main object is simplification of the circuit, and there is no idea of controlling the level of distortion correction in order to improve observability of the user.
Patent Document 1: Japanese Patent No. 3047581 “Image Distortion Correction Method and Device” distortion correction with a line memory
Patent Document 2: Japanese Patent No. 3451017 “Image Processing Method and Device” distortion correction in horizontal and vertical two stages
Also with an object of correcting distortion of a digital camera, digital distortion correction is proposed (refer to Non-Patent Document 8). According to the digital distortion correction, it is possible to change the level of distortion and to improve the accuracy of the distortion correction by converting parameters for the distortion correction. In this case, however, the object is to uniformly reduce distortion point-symmetrically distributed about an optical axis.
Non-Patent Document 8: “Digital Image Correcting Method for Digital Camera”, Ricoh Technical Report No. 31 (December, 2005) http://www.ricoh.co.jp/about/business_overview/report/31/pdf/A3114.pdf
3. Distortion Correction/Image Conversion of Onboard Camera
Also as to an onboard camera, there are various proposals for facilitating visual recognition of the driver.
Patent Document 3 proposes distortion correction preserving a feeling for the distance by supplying loose distortion in the longitudinal and lateral directions on the assumption that complete correction of distortion is not necessarily advantageous for improvement of recognizability (particularly the feeling for the distance) of the driver. The principal object of this proposal is the improvement of the recognizability, and there is no idea of preserving the angle of view. Further, only loose conversion of distortion is described, and there is no concept of changing the level of distortion correction by dividing an image into regions.
Patent Document 3: Japanese Unexamined Patent Publication No. 2003-123064 “Image Conversion Device and Image Pickup and/or Display Device provided with the Same”
Patent Document 4 proposes a technique of extracting and enlargedly displaying a hitch portion in a vehicle having the hitch for pulling a trailer. This proposal proposes a technique of simultaneously displaying the hitch portion and a wide-range angle image as different images, and there is no concept of controlling the level of distortion correction.
Patent Document 4: Japanese Patent No. 3483143 “Drive Assistance Device”
Patent Document 5 proposes a technique of partially extracting an image of a noted region and simultaneously correcting distortion in a camera picking up an image at a wide angle of not less than 90 degrees. This proposal is equivalent to a technique of extracting an image after uniformly performing distortion correction, and there is no concept of controlling the level of distortion correction.
Patent Document 5: Japanese Patent No. 3446277 “Ambient Situation Display for Vehicle”
Patent Document 6 proposes a technique of optically supplying loose distortion correction by employing a prismatic optical member. Also in this proposal, only a technique of loosely changing distortion is described, and there is no concept of changing the level of distortion correction by dividing an image into regions. Further, the level of distortion correction cannot be changed after designing the optical system due to the optical distortion correction, and there is no concept of adjusting the correction level in response to the loaded state on the vehicle.
Patent Document 6: Japanese Patent No. 3446991 “Monitor”
In order to render the attitude of a vehicle and the ambient situation easily graspable, a method of virtually converting the position and the angle of a camera is proposed.
Patent Document 7 proposes a technique of reducing the difference between appearances of regions close to and far from the vehicle respectively by virtually converting the mounting height of a rear camera of the vehicle and reducing a reflection of the bumper forming an unnecessary region in the visual field by virtually converting the mounting position of the camera in the anteroposterior direction.
Patent Document 7: Japanese Patent No. 3624769 “Image Conversion Device for Rear Monitor for Vehicle”
In a back camera (refer to Non-Patent Document 9) employed as an option of a domestic automobile maker or Patent Document 8, the camera is virtually directed vertically downward, thereby recognizably displaying the positional relation between parking lines on a road and the user's vehicle. The principal objects of these proposals are improvement of recognizability, and the angle of view is sacrificed.
Non-Patent Document 9: Suzuki Accessory for Every Wagon Select View Backeye Camera http://www.suzuki-accessory.jp/every/navi/99020_b14.html
Patent Document 8: Japanese Patent No. 3286306 “Image Forming Device, Image Forming Method”
In order to correctly display side extensions and an expected course line, accuracy in camera mounting on the vehicle body is required. The accuracy of camera mounting is important also in a high-function camera performing the aforementioned coordinate conversion of images. A technique of adjusting and inspecting mounting of the camera is also important.
Patent Document 9 proposes a technique of comparing a reference test pattern set on a prescribed position in front of a vehicle and a determination pattern displayed on a prescribed position of a shot image in order to indicate whether or not the angle of a mounted camera is within a range allowing fine adjustment by image conversion or a method of adjusting the angle to the operator. This Patent pays attention to only angle adjustment of the camera, and there is no concept of adjusting a distortion coefficient.
Patent Document 9: Japanese Patent No. 3565749 “Test Method for Image Pickup Direction of Onboard Camera and Test Device Thereof”
Patent Document 10 proposes a technique of comparing a test pattern set on a prescribed position in front of a camera and an ideal image pickup position for the test pattern calculated from a reference position decided as a design value with respect to a vehicle with each other in order to correct the mounting position and the angle of the camera and deviation of a lens optical axis. While this Patent proposes calculation of camera parameters including a distortion coefficient from deviation between an actual image pickup position and the ideal image pickup position for the test pattern, there is no proposal for adjusting the distortion coefficient.
Patent Document 10: Japanese Patent No. 3632563 “Image Positional Relation Correcting Device, Steering Assistance Device comprising the Image Positional Relation Correcting Device, and Image Positional Relation Correcting Method”
As hereinabove described, the onboard camera generally has large lens distortion due to the employment of a wide-angle optical system. It is common to set a camera position so that the vehicle body is reflected in a part of an image as shown in FIG. 3, so that obstacles close to the vehicle body and the user's vehicle can be grasped at the same time. In the example of FIG. 3 showing an image of a car park, a rear bumper (denoted by 100 in FIG. 3) is reflected as apart of the vehicle body. While the vehicle body must be partially reflected, it follows that the vehicle body occupies a large region of the image if lens distortion is remarkable. In addition to the lens distortion, the mounting position of the camera is frequently offset from the center of the vehicle, and it is further difficult to detect the optimum camera position.
In distortion correction of a digital system, it is possible to generally completely correct lens distortion. FIG. 4 shows an image after distortion correction. In this case, the end of the vehicle body is hardly distorted, and such a disadvantage that the vehicle body is excessively largely reflected is dissolved. However, it follows that diagonal information is lost due to the solution of the distortion. The diagonal information denotes information on regions on both sides of the upper visual field of the camera, and corresponds to information on the upper right and upper left portions in FIG. 3. For example, information of the vehicle on the upper right of FIG. 3 is lost, as shown by broken lines in FIG. 4.
Whether or not the diagonal angle of view is important depends on the setting of the camera with respect to the vehicle body. The upper visual field of the camera picks up an image of the road surface if the camera has a large angle of depression. In this case, the road surface, having a possibility of including obstacles such as persons and vehicles interfering with the user's vehicle, is an important region. It is unpreferable that the diagonal information is lost due to the distortion correction. If the camera has a small angle of depression, the upper visual field of the camera is above the road surface. In this case, there is a small possibility that obstacles interfering with the user's vehicle are present, and the driver can directly visually recognize such obstacles in general. In the case of such camera setting, it is possible to apply distortion correction by attaching more importance to distortion than the diagonal information.
An object of the present invention is to provide a drive assistance device capable of dividing an input image into a plurality of regions and varying distortion correction level with the regions when performing lens distortion correction on the input image.